Systems and methods for a multi-tenant system providing virtual data centers in a cloud configuration

ABSTRACT

An automatically provisioned virtual private data center provided on a cloud network of multiple virtual private data centers that reside in one or more physical data centers and may migrate between multiple physical data centers. The VPDC is a encapsulated virtual structure provisioned according to customer selected options within a virtual data center having one or more foundation nodes. Different service profiles are provided in a provisioning interface to the customer, each defining several configuration specifics that are used to automatically provision a VPDC for the customer.

BACKGROUND

Enterprise customers, such as businesses, non-profits, governments,etc., continue to rely on information technology and networkarchitectures as their communication and productivity infrastructure.With the continued increase in public network bandwidth andavailability, these IT and network services may be provided by outsidecompanies, which leverage shared resources and expertise to provide agreater cost-savings to subscribing customers.

These outside companies may provide Virtual Computing environments,which may include a virtual component for nearly every conceivablephysical component. Virtual disks, virtual processors, virtual LANS,etc. All of these virtual elements may be run on large physicalcounterparts, capable of efficiently and cost-effectively servingmultiple virtual version (e.g., multiple virtual machines may run on asingle large server). Further, by servicing multiple customers, not onlyis the equipment more cost-effective (e.g., as compared to each customerpurchasing smaller machines individually), but the total resourcesneeded is reduced. While each individual customer would need to plan forpeak usage, a shared system may need only prepare for the aggregatepeak, which may be smaller by mismatches between peak usage. Forexample, time differences, demographic differences, product releasetiming, and any number of other things may allow one customer's peak toalign with other customers' lulls, providing less variance in usagerates.

Thus, there exists a need in the art for greater distribution, greatervirtualization, and greater efficiency in provisioning, maintenance, andcustomer control/management tools.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a network of physical data centers as cloud sites,according to one example embodiment of the present invention.

FIG. 1B illustrates an organizational structure of cloud sitecomponents, according to one example embodiment of the presentinvention.

FIG. 2 illustrates a Virtual Private Data Center structure, according toone example embodiment of the present invention.

FIG. 3 illustrates an example method, according to one exampleembodiment of the present invention.

DETAILED DESCRIPTION

The present invention provides a modularized Virtual Private Data Center(VPDC) structure within a cloud of various Physical Data Centers (PDC)for providing data services to a plurality of distinct customers. Theterm “cloud computing” may carry various meanings in the industry, butexample embodiments of the present invention relate to two key aspectsof the cloud arrangement. First, abstracting as much of the technicaldetails away from the end user and into the backend structure. Second,allowing those abstracted technical details to be implemented in anynumber of physical locations, e.g., seamlessly moving from PDC to PDC.

FIG. 1A illustrates one example embodiment of a cloud-based virtualdatacenter network. Each cloud site (e.g., 110 a to 110 g) may include astandard set of hardware. Alternatively, each cloud site may have adifferent set of hardware, but each set may be configurable to provide astandard set of resources as a provisioned VPDC. Hardware may includeservers, routers, firewalls, SANs, and any number of other hardwaredevices. Each site may have a different quantity of provisionalresources (e.g., 110 d), but may provide at least one foundation pointof deployment (e.g., as discussed below) with sufficient resources toprovision at least on standardized VPDC.

Each cloud site may include a site manager 110, which may include thehardware and software to provision, monitor, and maintain the variousVPDCs located within the cloud site. The site manager may reside on anindependent server, or may be incorporated into the servers used toprovision VPDCs. The site manager 110 may also be responsive forfacilitating the moving of VPDCs from one cloud site to another. Shoulda site fail, or become overused, VPDCs may migrate to entirely differentsites. By fixing the structure of the VPDCs abstractly, all of theconfiguration and service data may be transferred to another site, andrepositioned at the new site (on identical hardware or differenthardware). Each VPDC may be provisioned according to a service levelwith customizable options. Each manager may provision certain resourcesto meet the specifications of the provisioning requirements, and controlaccess to the VPDC for an assigned user.

Cloud sites are multi-tenant sites. Only one user-entity (e.g., a singleperson, a single company, a single association, a group of relatedentities) may be assigned to a VPDC, in a similar fashion as a physicalPDC that is assigned to only one user-entity (e.g., a university). Eachcloud site may then have a plurality of VPDCs provisioned within one ormore foundation PODs. A single user-entity may have multiple VPDCsassigned to them. While additional resources may be provisioned within acustomer's single VPDC, some customers may require multiple VPDCs. Thecloud site contains various resources, such as compute, storage,network, etc., which may be provisioned according to specificationsprovided to the site manager 110, and the foundation POD services(discussed below).

Example cloud sites may include one or more Foundation PODs to comprisethe Physical Data Center 140. These embodiments of the present inventionprovide “PODs,” points of deployment, for use in tenant provisioning ofVPDCs. The Foundation PODs may include a plurality of Service PODs,which may include data structures that store configuration data for eachservice type provided by the cloud representing all levels of availablefor use in a VPDC. FIG. 1B may illustrate one of the PDCs, which may beorganized into several Points of Deployment (PODs). Each Foundation PODmay include several Service PODs, while each Virtual Data Center (VDC)may include several Compute PODs. The VDC may include (e.g., via theseveral Compute PODs) the resources that may be provisioned for acustomer, such as servers and processor throughput, data drives anddatabases, network links and bandwidth, etc. The Foundation POD maycontain Management, Network, Storage, and Virtual Services PODs whichmay provide the VDCs with all the services required to provide networkconnectivity 280, redirection 211, outside firewall 220, tierconnectivity to the SAN 215, performance metrics 240, and availabilitymetrics 245. The VDC may consist of all the Compute PODs which mayprovide the services required for application deployment, e.g., 210,230, 235, 221, and 222. These structures (e.g., the Foundation POD andVDC) may comprise the raw material for the provisioning of a customer'sVirtual Private Data Center (VPDC). One or more VPDCs may then be carvedout within the VDC to capture all the configuration and managementdetails required within the Compute, Management, Network, Storage, andVirtual Services environments of the Cloud site tied to a singlecustomers cloud environment. Each cloud site (e.g., PDC) may containmultiple Foundation PODs, and each Foundational POD may contain multipleVDCs. Each cloud site may have different resources, but those resourcesmay be configured to provide one or more standardized Foundation PODs(e.g., FIG. 1B), with may be used to partition out one or more VPDCs.

The compute PODs, e.g., 165 may include of the VDC's computeenvironments and may include components such as clusters of ESX hostsand Storage Area Networks (SANs) for ESX host storage. The compute PODsmay include local networking, such as one or more Top of Rack Switches.While there may be several levels of security, the compute PODs mayinclude a server level firewall and file integrity monitoring services.From these resources, customers may be provisioned VMs with network,storage, and security rules from the ESX Hosts within a cluster. Thenetwork POD may consist of core network connectivity functions,including the management and provisioning of connectivity to the computePODs, to a Management Network, and to an outside, public network, e.g.the Internet.

The Management Services POD, e.g. 150 may consist of all the managersfor the different system elements and for the VPDC management servers(e.g., those that manage the compute, storage, security, and networkresources). The virtual services POD may perform such tasks as serverload balancing and provide another tier of firewall security, such as aperimeter firewall service. From these service PODs and compute PODs acustomer may be provisioned one or move VPDCs, which will include a setof VMs, security policy, and a network policy. By grouping the servicesall together into a single-VPDC-instance model, the design may bemodularly or discreetly contained, and thus may be able to move aroundwithin the VDC, Cloud Site, and between Cloud Sites with automatedreconfiguration of services. These services may be management and/orvirtual (e.g., URL, DNS, and Server Load Balancing). Additionally, allthe SLAs and historical data may be preserved with the VPDC as itmigrates around.

FIG. 2 illustrates a Virtual Private Data Center (VPDC) 200 according toan embodiment of the present invention. A VPDC 200 is a set of cloudresources (e.g., servers, firewalls, networking, storage, etc.)provisioned to support a virtual private data center of a single tenant.Thus, as applied to a single tenant, the resources of the cloud 100(FIGS. 1A and 1B) appear as illustrated in FIG. 2. The VPDC 200 mayinclude security resources 220, 221, 222; computational resources 210,230, 235; storage resources 215, and networking resources, e.g.,connections to network 280. Each of these resources may be provisionedby a tenant prior to deployment according to the tenant's selections ofservice level and options during a provisioning phase.

Security resources represent hardware and software firewalls, accesscontrol lists, hash checkers and file integrity monitoring systems,encrypters/decrypters, etc. The physical firewalls may be shared bymultiple tenants, and have virtual private firewalls allocated from thatshared resource. The cloud provider may manage resource allocation atboth the physical and virtual level to ensure optimal performance foreach customer, within their service level selections.

Computing resources represent servers, including processors, localmemory, and I/O connections. Most of a client's transactional data maybe stored in specialized high volume data drives, but the servers mayinclude several levels of memory, including long-term memory, forstoring configuration data used to provision virtual machines and otherservices related to computing resources. The compute resource mayrepresent the smallest unit of customer data. Each compute resource maybe dedicated to a single customer or single VPDC, and individualcomputes may not be shared. The physical server may be carved up intothese smaller compute resource units. These compute resources may thencontain the customer's workloads. Each compute resource may havesecurity, network, and storage resources allocated to it. Furthermore,the VPDC may define a grouping construct that takes each computeresource (along with its security, network, and storage resources) andtheir configurations together to make up a compute grouping. All ofthese computes may be defined by XML data and managed as a group.

Storage resources represent storage for both configuration data (thatwhich defines the virtual systems) and transactional data (datagenerated by the user). Storage may include a number of persistentmemory levels, including hard drives at the server level, solid statedrives and caches, large drive arrays for low-latency network storage,slower drive arrays for less latency-critical storage, and long-termsarchive and backup drives.

Networking resources represent routers, gateways, bridges, data-lines,switches, and hubs for organizing and facilitating network trafficwithin the cloud site. Some of these items may be used for the overallsystem, while others, or parts of others, may be provisioned to aspecific VPDC. Networking resources also represent connections to anoutside network (e.g., the Internet and/or various private/semi-privatenetworks). Connections to the outside may be measured in bandwidth ortraffic throughput. Network resources may include certain degrees of theoverall bandwidth, certain data rate maximums, certain number ofconnections, or some combination of these. Additionally oralternatively, network resources may be defined by a priority, (e.g., aQoS priority level discussed further below).

The cloud may include a performance monitor 240 and availability monitor245. These may server two broad functions. First, they may be used tomonitor the use (or overuse) of various resources by the cloudproviders. Who may then use the data for strategic decisions, such aspricing, new resource planning, new client acceptance planning, etc.Further, the performance monitor and availability monitor, may togetherperform one or more load balancing functions, ensuring the bestavailability for each customer, according to their service levelselections and current resource availability. Second, these monitors maybe used in assisting customers make selections for their VPDCs, anddetermine how efficient/effective their current VPDCs are handing theloads placed on them. Customers may see how much latency their lowservice level VPDC is experiencing, including delays, denial of service,and request-to-result computation times, and decide is upgrading is acost-effective option.

Within each of the VPDCs, there may be a collection of Network,Security, Storage, and Compute configurations and processes. Thesecollections are designed to support a multi-tenant deploymentinfrastructure in separate logical containers that support multiplelevels of service per deployment. The services may define, configure,provision, monitor, and/or control each of the resources discussedabove. These definitions may be broken into several pre-defined servicelevels. The multiple levels of service provide various levels of supportfor each of the service categories of the VPDC, e.g., Network SLAs,Security SLAs, and Storage SLAs Compute SLAs, QoS, and backup levels.FIG. 2 illustrates one example embodiment of a VPDC. Each VPDC mayconnect to a wide area network 280, e.g., a public network such as theInternet, via a first external firewall 220. Security element 220 mayinclude any number of other network security devices. On the other sideof security element 220 there may be a cluster of web compute tier 210,such as web accessible VMs and UIs. Further security 221 and 222 mayseparate the various compute tiers, such as the application compute tier230 and the database compute tier 235. Each of the various compute tiersmay interface with one or more storage area networks 215. Further, theexample system may include a performance monitor 240 that may indicatehow the system is performing, how well the quality of service is beingmaintained, how long user latencies are, and how over or under workedthe provisioned resources are. There may also be an availability monitor245, which may monitor for downtime, failed connections, and/or anyother system failures.

A VPDC may include a service profile to define one or more distinctservice levels and/or qualities. Example embodiments may provide severalseparate and distinguishable service levels for selection by the variousVPDCs. The VPDC may provide the logical base container for a customerbuild, because the VPDC may consist of all the configuration andprocesses required to support the application deployment architecture ofthe customer, that is, the virtual system carved out of the physicalresources for a particular user according to specified configurationdata where the particular user's proprietary and/or shared applicationsare executed. The VPDC may capture information about how varioussoftware architecture layers are connected together, the communicationrequirements between the layers, the security of the communication anddata, and the storage lifecycle requirements along with any retentionlevels.

Each VPDC may be further defined by the Service Profile selected by thecustomer. The ability to abstract various support, process, andconfiguration attributes from the overall application lifecycle processmay be necessary to provide the multi-tenant multiple-QoS levels thatthe application-deployment lifecycle(s) require. An example applicationdeployment lifecycle may consist of development/Test, Quality Assurance,and Production.

The configuration of the VPDC from selecting the service tier tospecific configuration of the actual application architecture iscaptured as meta-data in XML and enhanced through the middlewaresoftware level within the VPDC architecture. The manifest is then sentto the VPDC engine to provision the VPDC into the infrastructure wherespecific Network, Security, Storage, and Compute aspects are provisionedand monitored for adherence to SLAs, support, process, and billing.

A key architectural aspect in providing a cloud based VPDC experiencemay include the concept of Service Profiles. Example embodiments of thepresent invention are able to create separate distinguishable servicelevels and qualities, which may be accomplished by defining a minimumset of essential architectural components required to deploy a cloudcomputing infrastructure, and further define enhanced packages above theminimum. For example, three service levels may be provided, an essentialservice level, a balanced service level, and a premier service level.

TABLE 1 Example Service Tier Specifics: TYPE/ LEVEL ESSENTIAL BALANCEDPREMIER Security Port ACLs Perimeter firewall WAF Server tier firewallsFile integrity checker Storage 16 VM disks 8 VM disks per 4 VM disks perLUN per LUN LUN Network Best effort QoS Priority 4 QoS Priority 5 QoS

The example attributes of three example service levels illustrated intable 1 could be any number of other configurations, attributes, orquality levels. The examples for the security service type may becumulative, such that port access control lists (ACLs) may be used inall three service tiers, while the balanced tier also includes aperimeter firewall and a server tier firewall. Likewise, these servicesmay be provided at the premier service level with the added services ofa file integrity checker and a web application firewall. With regard tostorage, the size, speed, and specs of a logical unit number (LUN) maychange (e.g., increase/improve with technology), the service tiers maybe broken down by the number of Virtual Machine disks that are carvedout of a single LUN. Network service levels may follow industry standardQuality of Service (QoS) priority levels. For example, the essentialservice may provide only QoS priority 0, or a common “best effort”service level. Priority 4 may include a controlled load, and may be mostsuitable for applications such as streaming media or multi-playergaming. Priority 5 may include latency and jitter tolerances suitablefor applications such as interactive video/audio (e.g., IP telephony). Amore detailed example is shown below in Table 2, which is written in XMLand illustrates one example of an XML interface for automaticallyprovisioning VPDCs based on service levels.

In order to facilitate automatic provisioning of VPDC in a cloud format,a Cloud OS may be provided to envelope and harmonize a plurality ofindividual pieces, some pre-existing, some new to this application. TheCloud OS may provide all the business intelligence and processintegration logic for the VPDC. The root of the VPDC may include an XMLbased service catalog with meta-data that captures the business andprocess logic of the VPDC. This may be accomplished by developing aService Catalog that enables VPDC service differentiation to be capturedas XML metadata. Table 2 illustrates an example service catalog. Eachservice profile (e.g., Enterprise, Balanced, and Essential) includes thesame or similar attribute types, e.g., location, load balancer level,1-to-1 Network Address Translation, VM machine specs, etc., while eachof these vary in size and quality across the service profiles.

TABLE 2 Example XML Service Catalog Code. <serviceCatalog xmlns:xsi=siteschemaVersion=“1.0” xsi:schemaLocation=“URL” xmlns=“URL”> <productname=“Cloud2.0” version=“1”> <serviceProfiles> <serviceProfilename=“Enterprise”> <locationOptions name=“metro”> <option>SantaClara</option> <option>Boston</option> <option>London</option></locationOptions> <supportedOSes customerImage=“disabled”> <supportedOSname=“rhel4_64Guest” label=“RH Linux 4.x”  description=“Red HatEnterprise Linux 4.x 64bit” type=“linux” showOutsourcingOption=“no” /><supportedOS name=“rhel5_64Guest” label=“RH Linux 5.x”  description=“RedHat Enterprise Linux 5.x 64bit” type=“linux” showOutsourcingOption=“no”/> <supportedOS name=“winNetEnterprise64Guest” label=“Windows Server2003 (Ent)”  description=“Microsoft Windows Server 2003 (Enterprise64bit)” type=“windows”  showOutsourcingOption=“yes” /> </supportedOSes><loadBalancerMax>4</loadBalancerMax><nat1to1Available>true</nat1to1Available> <computeProfile> <sizename=“Small”> <cpu speed=“3” unit=“GHz” vCPU=“2”/> <memory size=“2”unit=“GB”/> <drs pool=“Enterprise − 2”/> <serverHA>enabled</serverHA></size> <size name=“Medium”> <cpu speed=“3” unit=“GHz” vCPU=“2”/><memory size=“8” unit=“GB”/> <drs pool=“Enterprise − 8”/><serverHA>enabled</serverHA> </size> <size name=“Large”> <cpu speed=“3”unit=“GHz” vCPU=“4”/> <memory size=“16” unit=“GB”/> <drspool=“Enterprise − 16”/> <serverHA>enabled</serverHA> </size></computeProfile> <networkProfile> <vlans Public=“1” Private=“4”> <vlantype=“Public” maxComputes=“0” id=“Outside Transit”/> <vlantype=“Private” maxComputes=“0” id=“Inside Transit”/> <vlantype=“Private” maxComputes=“123” id=“VM Tier01”> <serverGroup>VMTier01</serverGroup> </vlan> <vlan type=“Private” maxComputes=“123”id=“VM Tier02”> <serverGroup>VM Tier02</serverGroup> </vlan> <vlantype=“Private” maxComputes=“123” id=“VM Tier03”> <serverGroup>VMTier03</serverGroup> </vlan> </vlans> <networkIO>100M Guarantee,Class=Medium</networkIO> <internetBandwidth>Zero Commit, 95th PercentileBurst Model</internetBandwidth> <applicationPriority pip=“enabled”/><sslOffloading ssl=“enabled” maxCert=“4” customerCert=“true”/><serverLB> <loadBalancing>enabled</loadBalancing><poolingInterval>true</poolingInterval> <persistence> <optionval=“Source IP” default=“yes”/> <option val=“Active Cookie”default=“no”/> <option val=“Passive Cookie” default=“no”/></persistence> </serverLB> <glbAcceleration glb=“no”maxServersPerDomainName=“4”/> </networkProfile> <storageProfile><dataStorage> <drives> <drive type=“boot” required=“yes”incremental=“no”> <size unit=“GB” min=“15” max=“15” incremental=“0”default=“15”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“C:\”/> <mountPoint type=“linux” defaultName=“/”/></mountPointOptions> </drive> <drive type=“data” required=“yes”incremental=“yes”> <size unit=“GB” min=“10” max=“500” incremental=“50”default=“10”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“D:\”/> <mountPoint type=“linux” defaultName=“/data01”/></mountPointOptions> </drive> <drive type=“data” required=“no”incremental=“yes”> <size unit=“GB” min=“50” max=“500” incremental=“50”default=“50”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“E:\”/> <mountPoint type=“linux” defaultName=“/data02”/></mountPointOptions> </drive> <drive type=“data” required=“no”incremental=“yes”> <size unit=“GB” min=“50” max=“500” incremental=“50”default=“50”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“F:\”/> <mountPoint type=“linux” defaultName=“/data03”/></mountPointOptions> </drive> </drives> </dataStorage> <performance>3Tier ILM</performance> <accessMethod>Block Based SAN</accessMethod><availability>Dual Path Redundant SAN</availability><primaryDataProtection>SNAP copies every 4 hours, 36 hourretention</primaryDataProtection> <secondaryDataProtectiondescription=“Backup retention”> <retention length=“1” unit=“month”default=“yes”/> <retention length=“3” unit=“month”/> <retentionlength=“6” unit=“month”/> <retention length=“12” unit=“month”/><retention length=“36” unit=“month”/> <retention length=“84”unit=“month”/> </secondaryDataProtection> </storageProfile><securityProfile> <perimeterFirewall enabled=“true”>Enterprise GradeVirtual Firewall (Enterprise Resource Classes) </perimeterFirewall><serverTierFirewall enabled=“true”>Enterprise Grade StatefulFirewall</serverTierFirewall> <intrusionDetectionSystem enabled=“true”> VMsafe Compatible Virtual IPS</intrusionDetectionSystem> <securityACLsenabled=“true”>Configurable Port Profile ACL per servertier</securityACLs> <fileIntegrityMonitoringenabled=“true”>Enabled</fileIntegrityMonitoring> <urlFilteringenabled=“true”>Enabled</urlFiltering> <waf> <learningMode days=“60”/><learningMode days=“90”/> <learningMode days=“120”/> </waf></securityProfile> </serviceProfile> <serviceProfile name=“Balanced”><locationOptions name=“regional”> <option>North America</option><option>Europe Middle East Africa</option> <option>Asia</option></locationOptions> <supportedOSes customerImage=“disabled”> <supportedOSname=“rhel4_64Guest” label=“RH Linux 4.x” description =“Red HatEnterprise Linux 4.x 64bit” type=“linux” showOutsourcingOption=“no” /><supportedOS name=“rhel5_64Guest” label=“RH Linux 5.x” description =“RedHat Enterprise Linux 5.x 64bit” type=“linux” showOutsourcingOption=“no”/> <supportedOS name=“winNetEnterprise64Guest” label=“Windows Server2003 (Ent)” description=“Microsoft Windows Server 2003 (Enterprise64bit)” type=“windows” showOutsourcingOption=“yes” /> </supportedOSes><loadBalancerMax>1</loadBalancerMax><nat1to1Available>true</nat1to1Available> <computeProfile> <sizename=“Small”> <cpu speed=“3” unit=“GHz” vCPU=“1”/> <memory size=“2”unit=“GB”/> <drs pool=“Balanced”/> <serverHA>enabled</serverHA> </size><size name=“Medium”> <cpu speed=“3” unit=“GHz” vCPU=“1”/> <memorysize=“4” unit=“GB”/> <drs pool=“Balanced”/> <serverHA>enabled</serverHA></size> <size name=“Large”> <cpu speed=“3” unit=“GHz” vCPU=“2”/> <memorysize=“8” unit=“GB”/> <drs pool=“Balanced”/> <serverHA>enabled</serverHA></size> </computeProfile> <networkProfile> <vlans Public=“1”Private=“1”> <vlan type=“Public” maxComputes=“0” id =“Outside Transit”/><vlan type=“Private” maxComputes=“123” id=“VM Tier01”> <serverGroup>VMTier01</serverGroup> <serverGroup>VM Tier02</serverGroup><serverGroup>VM Tier03</serverGroup> </vlan> </vlans> <networkIO>100MGuarantee, Class=Medium</networkIO> <internetBandwidth>Zero Commit, 95thPercentile Burst Model</internetBandwidth> <applicationPrioritypip=“enabled”/> <sslOffloading ssl=“enabled” maxCert=“1”customerCert=“true”/> <serverLB> <loadBalancing>enabled</loadBalancing><poolingInterval>true</poolingInterval> <persistence> <optionval=“Source IP”/> <option val=“Active Cookie”/> <option val=“PassiveCookie”/> </persistence> </serverLB> <glbAcceleration glb=“disabled”maxServersPerDomainName=“0”/> </networkProfile> <storageProfile><dataStorage> <drives> <drive type=“boot” required=“yes”incremental=“no”> <size unit=“GB” min=“15” max=“15” incremental=“0”default=“15”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“C:\”/> <mountPoint type=“linux” defaultName=“/”/></mountPointOptions> </drive> <drive type=“data” required=“yes”incremental=“yes”> <size unit=“GB” min=“10” max=“500” incremental=“50”default=“10”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“D:\”/> <mountPoint type=“linux” defaultName=“/data01”/></mountPointOptions> </drive> <drive type=“data” required=“no”incremental=“yes”> <size unit=“GB” min=“50” max=“500” incremental=“50”default=“50”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“E:\”/> <mountPoint type=“linux” defaultName=“/data02”/></mountPointOptions> </drive> <drive type=“data” required=“no”incremental=“yes”> <size unit=“GB” min=“50” max=“500” incremental=“50”default=“50”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“F:\”/> <mountPoint type=“linux” defaultName=“/data03”/></mountPointOptions> </drive> </drives> </dataStorage> <performance>2Tier ILM</performance> <accessMethod>Block Based SAN</accessMethod><availability>Dual Path Redundant SAN</availability><primaryDataProtection>SNAP copies every 8 hours, 36 hourretention</primaryDataProtection> <secondaryDataProtectiondescription=“Backup retention”> <retention length=“1” unit=“month”/><retention length=“3” unit=“month”/> <retention length=“6”unit=“month”/> <retention length=“12” unit=“month”/> <retentionlength=“36” unit=“month”/> <retention length=“84” unit=“month”/></secondaryDataProtection> </storageProfile> <securityProfile><perimeterFirewall enabled=“true”>Enterprise Grade Virtual Firewall(Balanced Resource Classes) <defaultRules> <firewallRule id=“10”source=“public” sourcePort=“http” destination=“VM Tier01”destinationPort=“http” protocol=“tcp” action =“permit” log=“no”/><firewallRule id=“20” source=“public” sourcePort=“https” destination=“VM Tier01” destinationPort=“https” protocol=“tcp” action =“permit”log=“no”/> <firewallRule id=“30” source=“public” sourcePort=“ssh”destination=“VM Tier01” destinationPort=“ssh” protocol=“tcp”action=“permit” log=“no”/> <firewallRule id=“40” source=“public”sourcePort=“any” destination=“any” destinationPort=“any” protocol=“tcp”action=“deny” log=“yes”/> </defaultRules> </perimeterFirewall><serverTierFirewall enabled=“true”>VMsafe Compatible VirtualFirewall</serverTierFirewall> <intrusionDetectionSystemenabled=“false”/> <securityACLs enabled=“true”>Configurable Port ProfileACL per virtual port group</securityACLs><fileIntegrityMonitoring>disabled</fileIntegrityMonitoring><urlFiltering>disabled</urlFiltering> <waf/> </securityProfile></serviceProfile> <serviceProfile name=“Essential”> <locationOptionsname=“global”/> <supportedOSes customerImage=“disabled”> <supportedOSname=“rhel4_64Guest” label=“RH Linux 4.x” description=“Red HatEnterprise Linux 4.x 64bit” type=“linux” showOutsourcingOption=“no” /><supportedOS name=“rhel5_64Guest” label=“RH Linux 5.x” description=“RedHat Enterprise Linux 5.x 64bit” type=“linux” showOutsourcingOption=“no”/> <supportedOS name=“winNetEnterprise64Guest” label=“Windows Server2003 (Ent) ” description=“Microsoft Windows Server 2003 (Enterprise64bit)” type=“windows” showOutsourcingOption=“yes” /> </supportedOSes><loadBalancerMax>0</loadBalancerMax><nat1to1Available>false</nat1to1Available> <computeProfile> <sizename=“Small”> <cpu speed=“1.5” unit=“GHz” vCPU=“1”/> <memory size=“1”unit=“GB”/> <drs pool=“Essential”/> <serverHA>Best Effort</serverHA></size> <size name=“Medium”> <cpu speed=“1.5” unit=“GHz” vCPU=“1”/><memory size=“2” unit=“GB”/> <drs pool=“Essential”/> <serverHA>BestEffort</serverHA> </size> <size name=“Large”> <cpu speed=“3” unit=“GHZ”vCPU=“1”/> <memory size=“2” unit=“GB”/> <drs pool=“Essential”/><serverHA>Best Effort</serverHA> </size> </computeProfile><networkProfile> <vlans Public=“1” Private=“0”> <vlan type=“Public”maxComputes=“253” id=“VM Tier01”> <serverGroup>VM Tier01</serverGroup></vlan> </vlans> <networkIO>100M Guarantee, Class=Medium</networkIO><internetBandwidth>Zero Commit, 95th Percentile BurstModel</internetBandwidth> <applicationPriority pip=“disabled”/><sslOffloading ssl=“disabled” maxCert=“0”/> <serverLB><loadBalancing>disabled</loadBalancing><poolingInterval>true</poolingInterval> <persistence> <optionval=“Source IP”/> <option val=“Active Cookie”/> <option val=“PassiveCookie”/> </persistence> </serverLB> <glbAcceleration glb=“disabled”maxServersPerDomainName=“0”/> </networkProfile> <storageProfile><dataStorage> <drives> <drive type=“boot” required=“yes”incremental=“no”> <size unit=“GB” min=“15” max=“15” incremental=“0”default=“15”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“C:\”/> <mountPoint type=“linux” defaultName=“/”/></mountPointOptions> </drive> <drive type=“data” required=“yes”incremental=“yes”> <size unit=“GB” min=“10” max=“500” incremental=“50”default=“10”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“D:\”/> <mountPoint type=“linux” defaultName=“/data01”/></mountPointOptions> </drive> <drive type=“data” required=“no”incremental=“yes”> <size unit=“GB” min=“50” max=“500” incremental=“50”default=“50”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“E:\”/> <mountPoint type=“linux” defaultName=“/data02”/></mountPointOptions> </drive> <drive type=“data” required=“no”incremental=“yes”> <size unit=“GB” min=“50” max=“500” incremental=“50”default=“50”/> <mountPointOptions> <mountPoint type=“windows”defaultName=“F:\”/> <mountPoint type=“linux” defaultName=“/data03”/></mountPointOptions> </drive> </drives> </dataStorage> <performance>1Tier ILM</performance> <accessMethod>Block Based SAN</accessMethod><availability>Dual Path Redundant SAN</availability><primaryDataProtection>SNAP copies every 24 hours, 36 hourretention</primaryDataProtection> <secondaryDataProtection> <retentionlength=“1” unit=“month”/> <retention length=“3” unit=“month”/><retention length=“6” unit=“month”/> <retention length=“12”unit=“month”/> <retention length=“36” unit=“month”/> <retentionlength=“84” unit=“month”/> </secondaryDataProtection> </storageProfile><securityProfile> <perimeterFirewall enabled=“false”/><serverTierFirewall enabled=“false”/> <intrusionDetectionSystemenabled=“false”/> <securityACLs enabled=“true”>Configurable Port ProfileACL per server a N1k </securityACLs><fileIntegrityMonitoring>disabled</fileIntegrityMonitoring><urlFiltering>disabled</urlFiltering> <waf/> </securityProfile></serviceProfile> </serviceProfiles> </product> </serviceCatalog>

Via a user interface to present the various user selectable options, anXML design file may be constructed for the automatic provisioning of theVPDC. Example embodiments of this portion of the Cloud OS are describedin U.S. patent application Ser. No. 12/646,591, filed on Dec. 23, 2009,the entire contents of which are expressly incorporated herein byreference. VPDCs may be provisioned via a portal manager constructed XMLdesign file, similar to that described in the incorporated reference.Portions of the design may be customizable within the service levelselection (e.g., a “Balanced” service level may provide VMs with storagebetween 50 and 500 GB, leaving the user to select the desired level),while other portions may be fixed by the selection of the service level.Options may also exist for automatic level selections of resources. Forexample, customers may pay some fractional amount for each GB ofstorage, with a minimum of 50 per VM and a maximum of 500, and havethose GBs provisioned in real-time, based on usage rates of thecustomer.

As part of the provisioning, each VPDC may be given its own set ofnetwork capabilities per service profile. For example, a VPDC that hasthe highest level of QoS SLA from the data center edge router all theway down to the 1 Gbps network for each Virtual Machine may be providedwith a QoS level of 5. The network QoS may be enhanced with privateMultiprotocol Label Switching (MPLS) network connectivity that providesend-to-end QoS across the network. Most clouds are only accessible overthe public Internet which offers no QoS beyond priority 0, best effort.Using the models described herein, a MPLS connection may be made withthe customer(s), which may allow for QoS levels as a service within VPDCprovisioning. With public Internet, Secure Socket-Layer (SSL) VirtualPrivate Network (VPN) may provide a high QoS within a tunneledconnection for added security. Additionally, at the highest level,Global Load Balancing (GLB) and Server Load Balancing (SLB) capabilitiesmay be provided for some number of server pools, e.g., eight. GLB mayenable site selection to ensure the best cloud VDC will be able to meetthe requirements of the server request.

At the lowest level, e.g., a QoS level of 1, best effort network servicemay be provided. This may leave room for multiple levels of servicebetween, e.g., 2, 3, and 4. Level 1 may primarily be a whatever capacityis left unused level of service, or may alternatively specify someminimal levels of service.

As part of the provisioning, each VPDC profile may receive security frommultiple levels of available security QoS. At the lowest end of servicelevels, it may be that only virtual firewalls and Access Control Lists(ACLs) are provided. At a higher level of service, the virtual Firewallmay be enhanced for more flexibility, and additional features may beadded, such as an intrusion detection alarm system (IDAS). For oneexample embodiment, the highest level of security QoS may include aphysical firewall, an Intrusion Protection System (IPS), a FileIntegrity Monitor, and one or more Web Application Firewall(s) (WAFs)may be provided.

The security profile design may also take into account that often, atthe lowest level, the test/develop environment may be flat and typicallyrequire some basic perimeter firewall capability. At the next higherlevel, a requirement to support multiple tiers, for example web,application, database or front-office/back-office application deploymentmethodologies may require the security profiles to enable publicinterface to the first tiers of services through a perimeter firewall.The next level up may include a deep packet inspection capable firewallbetween the web application tier and the application tier, and betweenthe application tier and the database tier (e.g., 221 and 222 of FIG.2). This may require understanding the communication flows between eachtier and layer. Such as specific application architecture and thelatency requirements of certain applications and configurations.Communication flow information may also include which security ports toopen for these inter-tier security elements, along with anidentification of what zones each customer wants to isolate from othertraffic.

In order to achieve this added inter-tier security one or more servicelevels may create a separate VLAN per tier. This may require inter-tiertraffic to flow back to the core switch and thus get inspected by afirewall. To do this with separate hardware may add significant latencyto customer traffic, and added stress on the overall networkperformance. However, by structuring the segregations as Virtual LANsand providing the entire VPDC with virtualization, all the traffic canstay within the same network domain. Thus, for one or more servicelevels a single network domain may be established that is segmented intosome number (e.g., 3) of port groups (e.g., a logical container for eachtier). This may provide a tier segmentation without significant addedlatency. A Server Tier firewall may then be defined to have a policy foreach tier and provide a security boundary between each tiers.

As part of the provisioning, each VPDC profile may include a level ofdata storage. Information Lifecycle Management (ILM) may enable data towaterfall down to lower-cost data stores as the data access on thesefiles decreases. At the lowest level of service, it may be that only onetier of ILM is available. At the highest level of service, it may bethat 3 tiers are available. For example, in the Essential serviceprofile there may be a single tier (e.g., tier-3 SATA storage). Movingup a service level, the Balanced service profile may initially use atier-2 storage that include 10 k rpm fibre channel drives, while lessfrequently used data may automatically migrate down to the tier-3 SATAstorage system. The Premier service profile may initially use a tier-1storage including 15 k rpm fibre channel drives, while less frequentlyused data may automatically migrate down to the tier-2 storage, whileeven less frequently used data may automatically migrate down to thetier-3 storage.

Further, each storage service level may provide a different level ofback-up service and/or retention time. For example, the highest level ofservice may provide fault-tolerant back-up for 24 months, while thelowest level may provide generic (e.g., single copy) back-up for only 4months. Each profile may also define a compute QoS that includesoperating system(s), applications, configurations, etc. The computeprofile may define how many Virtual Machines are available at any onetime, and how much execution throughput is available to each or the setof VMs.

After a user makes the desired selections to form an XML design file, anautomated provisioning method may begin. One example is found in FIG. 3.The example method may start the provisioning process at 301, which mayload a manifest created based on the XML design file (or alternativelymay load the XML design file itself to be used as the manifest), e.g.,at 304. Next, at 307, the example provisioning method may parse themanifest to pull all the variables specified in the manifest to be usedby subsequent provisioning subroutines. At this point the actualprovisioning subroutines may be called, but first, at 310, the examplemethod may call a save note function to save the initial stateinformation. This may allow for a persistent context, which may beresumed or recovered from, in the event of a process failure.

The example method may next create the storage at 313, which may includea multi-step process to create a volume, map it to the ESX hosts, andcreate a data store for the VPDC. Next, at 317, the network provisioningsubroutine may be executed, which may include a multi-step process tocheck device connectivity, check if requested VLANs exist, provision theVLANs, and establish/provision the port-profiles. Next, at 320, thesecurity provisioning subroutine may be executed, which may include amulti-step process to create security ACLS for one or more servicelevels (e.g., Essential VPDCs). For service levels with even moreprovisioned security, the security subroutine may provide ACE context,NAT, and perimeter firewall rules (e.g., for Balanced VPDCs).

Next, at 323, an AD and DNS build subroutine may create OUs, groups, andusers in a network directory service (e.g., Active Directory), alongwith creating a customer's domain name server zone. At 327, the examplemethod may create folders for the VPDC. This may be done with a numberof tools/services, e.g., an Open Source Software (OSS) Web Service andvSphere™. At 330, the example method may build one or more VirtualMachines using a Build VMs subroutine to provision one or more Windows®and/or Linux virtual machines. The VM build(s) may be checked (seriallyor in parallel) by confirming each result of the global build outputs,or by performing one or more verifying calculations. At 337 the examplemethod may finalize the provisioning operations, e.g., by deletinginstall files no longer needed, emailing the customer that the VPDC isready and providing usage information, and opening up the network foroutside traffic to the VPDC.

It should be understood that there exist implementations of othervariations and modifications of the invention and its various aspects,as may be readily apparent to those of ordinary skill in the art, andthat the invention is not limited by specific embodiments describedherein. Features and embodiments described above may be combined. It istherefore contemplated to cover any and all modifications, variations,combinations or equivalents that fall within the scope of the basicunderlying principals disclosed and claimed herein.

We claim:
 1. A provisioning system for a cloud computing network,comprising: a storage device, storing: a plurality of POD datastructures, each POD data structure storing parameter data for arespective service supplied by the cloud under a variety ofconfigurations supported by the service, a plurality of VPDC datastructures, each VPDC data structure storing parameter data extractedfrom the POD structures according to a respective level of serviceselected by a respective tenant, and a processor, executing a userinterface for engagement with a tenant of the cloud network, to presentdata of the VPDC data structures to a tenant and receive selections inresponse thereto, and further to provision a tenant VPDC within thecloud according to the selected VPDC configurations and service level.2. The system of claim 1, wherein to provision includes establishing aplurality of application tiers; wherein to provision further includesestablishing network services including a unique virtual local areanetwork (VLAN) for each application tier; wherein to provision furtherincludes creating separate port groups and assigning a unique port groupto each VLAN.
 3. The system of claim 2, wherein to provision furtherincludes establishing a firewall between each VLAN.
 4. A method ofprovisioning resources of a cloud computing network by a cloud tenant,comprising: retrieving, from storage, data of a plurality of VirtualPrivate Data Center (VPDC) configuration options, the VPDC configurationoptions storing parameter data extracted from POD data structures of thecloud network, the POD data structures storing parameter data for arespective service supplied by the cloud under a variety of serviceconfigurations, presenting the retrieved configuration options to thetenant, receiving selections in response to the presented configurationoptions, and installing a Virtual Private Data Center (VPDC) instancewithin the cloud network according to the configurations selected. 5.The method of claim 4, wherein the VPDCs are configured to migrate fromone cloud site to another cloud site within the cloud network.
 6. Themethod of claim 4, wherein the parameter data includes definingprocessor power, data storage, and network capacity.
 7. The method ofclaim 4, wherein a plurality of data storage tiers are available andwherein a lesser service level is associated with a lesser data storagetier and a greater service level is associated with a greater datastorage tier.
 8. The method of claim 7, wherein data associated with aVPDC associated with the greater service level is initially stored inthe greater data storage tier, and parts of the data that areinfrequently accessed are migrated to the lesser data storage tier. 9.The method of claim 4, wherein the installing includes establishing aplurality of application tiers; wherein the installing further includesestablishing network services including a unique virtual local areanetwork (VLAN) for each application tier; wherein the installing furtherincludes creating separate port groups and assigning a unique port groupto each VLAN.
 10. The method of claim 9, wherein the installing furtherincludes establishing a firewall between each VLAN.
 11. A method ofprovisioning a Virtual Private Data Center (VPDC) in a cloud network ofphysical data centers, comprising: providing a design user interface toa user; receiving configuration selections from the user, including aservice level selection; creating a design manifest based on theconfiguration selections; automatically provisioning a VPDC based on thedesign manifest, wherein the VPDC is provisioned in a cloud sitephysical data center configured to provide a plurality of VPDCs, andwherein the cloud site physical data center is organized with aplurality of service deployments and a plurality of compute-resourcedeployments, which together provide the VPDCs, based on configurationdata specified by each VPDC.
 12. The method of claim 11, wherein theVPDCs are configured to migrate from one cloud site physical data centerto another cloud site physical data center.
 13. The method of claim 12,wherein the plurality of service deployments and the plurality ofcompute-resource deployments are standardized at each cloud sitephysical data center.
 14. The method of claim 11, wherein substantialportions of the VPDC are defined by the service level selection.
 15. Themethod of claim 14, wherein the substantial portions include definingprocessor power, data storage, and network capacity.
 16. The method ofclaim 15, wherein a plurality of data storage tiers are available andwherein a lesser service level is associated with a lesser data storagetier and a greater service level is associated with a greater datastorage tier.
 17. The method of claim 16, wherein data associated with aVPDC associated with the greater service level is initially stored inthe greater data storage tier, and parts of the data that areinfrequently accessed are migrated to the lesser data storage tier. 18.The method of claim 11, wherein the provisioning includes establishing aplurality of application tiers; wherein the provisioning furtherincludes establishing network services including a unique virtual localarea network (VLAN) for each application tier; wherein the provisioningfurther includes creating separate port groups and assigning a uniqueport group to each VLAN.
 19. The method of claim 18, wherein theprovisioning further includes establishing a firewall between each VLAN.